SOF-8005: parse NWChem initial and final geometry - #165
Conversation
NwchemParser implemented none of IonicDataMixin's four geometry methods,
whose bodies are bare `pass`, so lattice_basis_to_poscar raised on None
and every NWChem job silently failed to publish its structures --
extract_structures runs app-agnostically for every modeling unit, and
extract_property swallows the exception into a log.error.
One extractor keyed on the "Output coordinates" block, plus four thin
delegates. Notes:
- The block header carries the units the input declared. test-001 is a
`geometry units au` run and prints no angstrom block at all, so the
units word is read rather than assumed; an angstroms block is taken
verbatim rather than rescaled through two Bohr radii that disagree in
the last digits.
- The cell is not ours to invent: made's calculate_padded_cell_simple_cubic,
the same convention every non-periodic material on the platform gets.
- BASIS in tests/fixtures/nwchem/references.py held test-001's printed
a.u. geometry mislabelled `"units": "angstrom"`. It was never used by
any test, so nothing caught it. Corrected and now asserted.
Also returns those structures as molecules. NWChem works in the finite
molecular picture, but Material defaulted is_non_periodic to False and
rupy cannot correct it -- it only ever sees the material's _id, never the
material. So a relaxed molecule came back as a periodic crystal, with
volume/density where inchi/inchi_key belong. The parser now answers for
itself, an explicit kwarg still wins, and `isNonPeriodic` is serialized
so the flag survives to the platform. Timur asked for exactly this on
exactly this file in express#55 (2020-12-08).
Verified on a real job once deployed: H2O, CUB, inchi 1S/H2O/h1H2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a7dd652 to
2a42ffd
Compare
_lattice_vectors was derived per-block, so initial and final got different boxes -- 5.72 A and 3.05 A for the same water molecule. The final one shrink-wrapped the relaxed geometry: made's padding factor is 2x the max pairwise distance, which for water leaves ~0.8 A of vacuum a side, and the two structures could not be compared. An optimization moves atoms inside a fixed box; it does not resize the box. The cell is now derived once, from the first geometry block, and both structures share it -- fixed-cell semantics, and the relaxed molecule sits in the roomier unrelaxed box instead of being wrapped tight. Smaller than what it replaces: _lattice_vectors no longer takes `last`. Still made's convention, unchanged; only how many times it is applied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cell was derived from the first geometry block alone, so a relaxation that expands a molecule left atoms outside the box. Reproduced: a diatomic going 0.2 -> 2.5 A landed at z = 4.0 and -1.0 in a 3.0 A box, which reads as extra fragments and turns inchi 1S/H2O/h1H2 into 1S/H2O.2H2/h1H2;2*1H -- silently wrong chemistry, not a visible failure. Still one cell for both, so the two structures stay comparable; it is now sized to whichever geometry needs more room. Contracting relaxations, the common case, are unaffected: the fixture still gives 5.72171236 for both, and the H2O acceptance job still gives 3.163849. Also drops four restatements of why the cell is shared, keeping the explanation where the logic is. Found by review: the problem was repetition, not volume -- express sits below this repo's docstring density. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| return self._lattice_vectors(text) | ||
|
|
||
| def final_lattice_vectors(self, text): | ||
| """Same cell as the initial structure. See `_lattice_vectors`.""" |
| continue | ||
| elements.append(element.group(1)) | ||
| coordinates.append([float(row.group(axis)) * to_angstrom for axis in ("x", "y", "z")]) | ||
| return elements, coordinates |
There was a problem hiding this comment.
Can we write a complex regular expression to extract this exact information?
Review on #165: parse with a complex regular expression built from primitives rather than cutting lines with find/slice and filtering in a loop. GEOMETRY_BLOCK_REGEX now spans header to last atom row and captures the rows in a group, composed from named primitives - the header, an element symbol, a tag, the dashed rule, a row template and a tempered gap that cannot cross into the next block. The trailing (?:row)+ bounds the table by itself, so the two find() calls and the manual slice to the following header are gone. Capturing (?P<element>[A-Za-z]+)\S* in the row makes ELEMENT_FROM_TAG_REGEX and its skip-on-no-match branch unnecessary: a row of some other table cannot match in the first place. _geometry_block drops from 30 lines to 12, with no loop and no ifs beyond the empty-match guard. Verified the bounding is unchanged: 13 blocks of 3 rows each in the multistep frequency log, 1 of 3 in the a.u. total-energy log, same elements and coordinates as before. Also drops the one-line docstrings from the four delegates, per review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tb-review blocker on #165. `(?:row\n)+` stops at the first non-row line, so a log truncated mid-table parses as a shorter molecule and nothing compared the two blocks. Reproduced: initial ['O','H','H'], final ['O','H'], serialized as H2O2. rupy publishes it, because extract_structures() gates only on `if initial and final:` - and this is a new failure mode, since on main these methods returned None. _geometry_blocks() now parses every block once and returns nothing when they disagree on which atoms are present, so both callers fail closed. It also hosts the cell derivation, which previously reached three levels into _geometry_block's return. Also from the review: - drop the `angstroms` verbatim branch. 1.889725989 * Constant.BOHR is 1.0000000162985654, so it bought 5e-8 A of a 3 A cell for a units special case in every block. - final_lattice_vectors aliases initial_lattice_vectors rather than repeating the body; a fresh dict per call, so no aliasing hazard. - declare is_non_periodic = False on BaseParser instead of leaving it an implicit getattr protocol. Material keeps the getattr, being built with parser=None in tests. - the shared cell is for comparability, not containment: a cell derived per structure always contains its own atoms. Centering a basis in the other block's cell is what put atoms outside. Docstrings said both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two flagged on the diff, plus the same kind elsewhere: the header above the regex primitives, the gloss on ELEMENT_REGEX, and the first line above is_non_periodic on NwchemParser. 14 comment lines down to 9. What survives explains regex mechanics that are not readable off the pattern - why the tag must start with a symbol, what the tempered gap prevents, what bounds the table - and the one coupling between _basis and _lattice_vectors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
||
| # test-001 is a single point, so its initial and final structures are the same one printed block. | ||
| # Its input declares `units au`, so the coordinates below are the printed ones converted to | ||
| # angstrom; they are then centered in the cell that made's convention derives for a molecule. |
There was a problem hiding this comment.
Checked this properly — the previous values were the wrong ones.
test-001's input declares geometry units au noautosym and the log header reads Output coordinates in a.u. (scale by 1.000000000 to convert to a.u.). The old fixture had 1.43042809 verbatim, which is the input's a.u. value, under "units": "angstrom". Read as angstrom that geometry gives an O-H of 1.81 A; the experimental water bond is 0.9572 A.
Multiplying the old values by BOHR and centering them in the derived cell reproduces the new ones to 8 decimals, so the change is exactly the missing unit conversion plus the centering.
Worth noting BASIS was declared on main and asserted by no test, so nothing had ever compared it against a parser — this branch is the first thing that did. a488316 adds test_bond_length_of_an_atomic_units_block_is_physical, which feeds test-001's printed rows through the parser and asserts 0.9572 A, so it fails if the conversion is ever dropped.
claude
Review on #165 called the BASIS coordinates suspicious. They are the right ones and the old ones were wrong, but nothing in the suite proved it - BASIS was declared on main and asserted nowhere, so it had never been compared to a parser at all. test_bond_length_of_an_atomic_units_block_is_physical feeds test-001's printed rows through the parser and asserts O-H = 0.9572 A, the experimental water bond length. Read as angstrom the same rows give 1.81 A, so the test fails if the a.u. conversion is ever dropped. The fixture note now points at it instead of asserting the conversion in prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
||
| # Take the edge from _lattice_vectors rather than deriving a second cell here, so the basis | ||
| # is centered in the very box that ships with it. | ||
| center = get_center_of_coordinates(coordinates) |
| return [] | ||
| return blocks | ||
|
|
||
| def _basis(self, text, last): |
There was a problem hiding this comment.
It needs to extract all structure entries and then choose by id (1; -1 as options)
CEO review on #165: parsers parse text into data structures; the cell and the centering are a separate thing, one layer up. Material.__init__ already post-processes parser output, so box_molecule lands beside it. Inert until the next commit: it fires only when the parser returns no lattice, and today every parser returns one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
geometry_blocks() converts units and nothing else; basis(text, index) picks a block by id, 0 initial and -1 final, which is the "extract all structure entries and then choose by id" comment. The cell derivation and the centering are gone, and with them the mat3ra-made import. Both lattice delegates are deleted rather than reworded. They returned the same invented cell, and IonicDataMixin declares them `pass` without ABCMeta, so unimplemented is None - which is what VASP already relies on by omitting initial_*. references.py BASIS goes back to plain converted coordinates. It held centered ones, which is what "Seems suspicious" was pointing at: a fixture recording that the parser had moved the atoms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The relocation is only worth anything if it changed no output, so assert the numbers the live job produced: edge 3.163848897 and O at [1.337365565, 1.554783897, 1.291226523], from web-app's final-structure-nwchem-optimized.json fixture. test-003 is that geometry translated to the origin, so the padding and the centering are what have to put it back - fed its own published coordinates the pipeline would be a fixed point and a missing centering step would pass. Verified against a488316 rather than asserted: the full serialized material for test-001, test-002 and test-003, initial and final, is byte-identical before and after the move. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverses the fixture half of 9313866. test-003 was the relaxed cluster geometry re-emitted in NWChem's column format - the only "NWChem output" in the repo that no NWChem ever printed, and opaque fixture data is what triggered this review in the first place. The relocation invariant does not need it: the full serialized material for test-001 and test-002, initial and final, is byte-identical to a488316, and the cluster's own numbers stay pinned where they belong, in web-app's final-structure-nwchem-optimized.json and the feature's HOMO/LUMO assertions. So the assertions move onto test-002, which is a real geometry optimization. Mutation strength is unchanged or better: reverting the centering fails one test, reverting the per-block sizing now fails two (test-002's final block alone gives 3.05 A, not 5.72), reverting the material.py hook fails three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TB-TEST-1. The restructure deleted test_shared_cell_fits_a_relaxation_that _expands and replaced nothing, and the reviewer showed what that cost: change material.py's initial branch to box_molecule(basis, [basis]) and the whole suite still passes. test-002's initial block is the larger of its two, so its padded edge already IS the shared edge - the expanding direction, the one that leaves atoms outside the box and corrupts the InChI, is unexercisable by any committed fixture. tests/unit/properties/test_utils.py covers it with synthetic bases: the cell is sized to the largest structure, every structure is centred inside the shared cell, and Material hands over every parsed basis rather than only the one it is centring. That last one is what fails under the reviewer's mutation. The three Material tests move out of the nwchem parser module and in beside the other Material tests; the manifest is keyed on test name, so the move is free. Three rather than the two asked for: leaving the third behind would have kept a Material test in a parser module for no reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cell TB-NAME-1: `basis` and `bases` are one letter apart at a call site that passes both. Now selected_basis and parsed_bases. TB-DOC-1/3: NwchemParser's missing lattice methods are load-bearing control flow - material.py reads None as "molecule, derive a box" - and nothing said so once the old docstrings went with the methods. One sentence on the class, where a reader wondering why the methods are absent will be. Not repeated at the guard. Also from the review: basis() returns None with no geometry block, so say `dict | None` as the rest of the file does; box_molecule's docstring trimmed and its Returns made truthful; the padded cell gets a named intermediate instead of a bare [0][0]; and the fixture comment drops the SHA it cited, which will not survive a squash-merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
a488316 swept black reflows and blank-line fixes into seven files no nwchem change touches. Restore them to main byte-for-byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…xtures Cuts the branch from 405 insertions over origin/main to 296. What went is prose and code; the data and the docstrings stayed. Code and prose: - the three Material tests merge into one, since all three read the same fixture, and the manifest loses two entries with them - test_converts_a_block_printed_in_atomic_units folds into the bond-length test, which asserts the same conversion more strictly - test_nwchem_structures_of_optimization goes, with its two basis constants. Its numbers are still pinned one layer up: the cell edge comes from test-002's initial block and the crystal coordinates from its final one, so the serialized-material test covers both - geometry_blocks() and box_molecule() keep the why-prose and lose the Args/Returns boilerplate; settings.py's regex comments keep their constraint and lose the restatement Data was not golfed. The unit fixtures print a real NWChem geometry block - header, column titles, dashed rule, column-aligned rows, atomic-mass tail - and the bond-length test carries test-001's three rows verbatim with both O-H distances asserted, because that is what the log contains. basis() keeps its Example, as its siblings have one, and the two parser delegates keep the full docstring the methods above them use. Coverage went up, not down: the truncation guard had no test at all and now has one. All four mutations still fail - centering, per-block sizing, the material.py guard, and the initial call site. The serialized material for test-001 and test-002 is byte-identical to 04f4d7d. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This PR (express)
NwchemParserimplemented none ofIonicDataMixin's four geometry methods, whose bodies are barepass. Solattice_basis_to_poscarraised onNone, and every NWChem job ever run silently failed to publish its structures —extract_structures()runs app-agnostically for every modeling unit, andextract_propertyswallows the exception into alog.erroron the compute node. That predates the relaxation work entirely.One extractor keyed on the
Output coordinatesblock, plus four thin delegates. Three things worth a reviewer's attention:tests/fixtures/nwchem/test-001is ageometry units aurun that prints no angstrom block at all — the plan's instruction to key on the literalOutput coordinates in angstromswould have returnedNonefor it, leaving that silent failure open for every au-geometry job. The units word is read rather than assumed. Anangstromsblock is taken verbatim rather than rescaled through two Bohr radii that disagree in the last digits.made'scalculate_padded_cell_simple_cubic, the same convention every non-periodic material on the platform gets. Initial and final therefore get different edges, as made would give them: 5.72171236 Å vs 3.05045924 Å for the H2O fixture. The basis takes its edge from_lattice_vectorsso it is centered in the box that ships with it.BASISintests/fixtures/nwchem/references.pywas wrong and unused. It held test-001's printed a.u. geometry tagged"units": "angstrom"— added in the original NWChem PR and never referenced by any test, so nothing caught it. Corrected to that geometry converted and centered, and now actually asserted.Tests: 22 passing (unit + integration), including a regression case for a numeric row that fits the column shape but has no element symbol — it is skipped rather than raising, because rupy would swallow the exception into a silently missing
final_structure.Pre-existing on
main, untouched: 11 failures intests/integration/parsers/test_structure.py,tests/unit/parsers/test_structure_parser.pyandtests/integration/properties/test_material.py, all espresso structure parsing. Confirmed by stashing every change and re-running.Part of SOF-8005 — [Standata] Relaxation SWF/WF for NWChem. One ticket, five repos; this is one of five PRs that land together.
total_energycomputed HOMO/LUMO on whatever geometry was uploaded — nothing in the NWChem workflows relaxed anything, andget_relaxation_by_application("nwchem")returned{}, so Toggle Relaxation silently no-opped. These PRs close the four gaps between NWChem and what VASP/Espresso already do.MaterialEvidence
Acceptance test
job-nwchem-relaxation-total-energy.featurepasses against a live cluster — takes the unrelaxedH2O.poscar, toggles relaxation in the real UI, runs one job:homo_energylumo_energyTolerance is 0.02 eV, not the 0.1 eV the other nwchem features use: relaxed and unrelaxed HOMO differ by only 0.053 eV, so 0.1 would pass on an unrelaxed molecule — precisely what this test exists to catch. LUMO differs by 0.341 eV and is the discriminating assertion.
The sharpest evidence is not in the assertions. The relaxation unit reported
total_energy = -2078.5728586862333and the total-energy unit-2078.5728586879750— agreeing to 2e-9 eV. The second unit reproduced the first's final energy, which it can only do by computing on the relaxed geometry; astart-from-scratch unit would have recomputed the original upload.Manual checks
restart nwchemwith nogeometryblock — read from a real job atjob.workflow.subworkflows[].units[].input[0].rendered.Optimization convergedstring was not read; job files are in object storage behind the authenticated API.Materialappears with a resolvablematerialId— still absent, and deploying express will not fully fix it.is_non_periodicappears nowhere in rupy andMaterial.__init__defaults it toFalse, so the relaxed molecule builds throughCrystalParser: cubic lattice type yes,isNonPeriodicno. Timur asked for exactly this on this exact file in express#55 (2020-12-08): "Are we able to retain the fact that this is an isolated molecule, rather than a periodic system…". Needs a rupy change; scoped out here and filed as a follow-up.{nwchem, 7.0.2, GNU, isDefaultVersion: true}.Geometry Relaxation.Frequency (nwchem)with no relaxation rendersstart nwchem, byte-identically to before.Review
Two
tb-reviewerpasses. The first found two blockers — localfile:tarball pins in web-app, and ajobHasParent ||half that would have rendered a deck NWChem cannot run — both fixed. The second, grounded against the precedent corpus (12,340 comments / 73 repos), found that Toggle Relaxation had become reachable onFrequency (nwchem)while that template still declared its own geometry: it would have relaxed, discarded the result, and reported frequencies for the original molecule, silently. Fixed here.Known and deliberately not fixed
is_non_periodic— above; needs a rupy change.isRelaxedwill readFalseon every NWChem relaxation once express is deployed._is_relaxed()derives it frompressureandatomic_forces, andNwchemParserimplements neither. Unreachable today because extraction fails first.wode Workflow.ts:352), not by what the workflow computes. wode#100 already pointed this way.🤖 Generated with Claude Code